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ABSTRACT 

Purpose - The study aimed to (1) review the literature to construct 
conceptual models that could guide instructional designers in 
developing problem/project-based learning environments while 
applying effective feedback strategies, (2) use the models to design, 
develop, and implement an online graduate course, and (3) assess 
the efficiency of the models. 

Method - Quantitative and qualitative data were gathered during 
the implementation of an online graduate-level course that was 
designed and developed using the two proposed conceptual models. 
Eleven students were enrolled in the course. Multiple sources of data 
and analytical strategies were used to collect, analyze, and interpret 
the data. 

Findings - The results show that the characteristics and strategies 
suggested in the proposed models supported student achievement 
in course projects and activities. The timely and frequent feedback 
allowed students to make necessary changes to their final project 
parts and improve performance. The motivating, informative/ 
specific feedback and feed-forward goals appeared to provide a 
reference for areas of student improvement. A balance of the three 
types of feedback, along with metacognitive questioning, supported 
student improvement. Because the frequency of feedback was 
dependent upon student request, those who needed more frequent 
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feedback during the acquisition of learning benefited from that, but 
those who did not need it received the moderate amount necessary 
to be successful. 

Significance - The study brings to light the importance of three 
types of formative feedback in problem or project-based learning. 
It offers two conceptual frameworks for instructional designers and 
curriculum developers. The models could be used to provide all 
forms of feedback including feed up, feedback and feed-forward to 
support student success in PBL learning environments. 

Keyword: Problem-based learning, Project-based learning, 
feedback and feed-forward, online learning. 


INTRODUCTION 


Problem-based learning (PBL) was first developed in 1960 in medical 
education with the aim of situating learning in a meaningful task, 
such as case-based instruction and project-based learning (Hmelo- 
Silver, 2004). As a student-centred learning environment, PBL was 
developed to bridge the gap between what is learned in school and 
its relevance for future professional practice (Wijnia, 2014). PBL 
starts with authentic, real-world problems or situations rather than 
the exposition of knowledge. Problems are used as tools to achieve 
the required knowledge and the problem solving skills necessary to 
eventually solve the problems (Barrows, 1996). Under the guidance 
of a facilitator, small groups of students work together to analyze the 
problems, formulate learning issues for further self-study, and then 
collaborate as a group to solve the problem. 

Project-based learning (PrBL) has often been viewed as synonymous 
with PBL. While there are some historical differences between 
the two approaches (Ertmer & Simons, 2006), scholars argue that 
problem-based learning is a type of project-based learning (Larmer, 
2014). A comparison of problem-based and project-based learning 
suggests that the essential components of project-based learning 
align with the following five PBL criteria: 1) it is a student-centred 
approach, 2) it starts with authentic and real world problems or 
tasks, 3) it requires collaboration and group work, 4) it has a 
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multidisciplinary orientation, and 5) it is guided by facilitators or 
tutors. The major differences between PrBL and PBL include: 1) 
project tasks are closer to professional reality, thus take a longer 
period of time to solve, 2) project tasks result in concrete products, 
3) embedded in the large problems or projects are smaller problems 
to be solved, and 4) projects are more directed to the application of 
knowledge rather than acquisition of knowledge (Perrenet, Bouhuijs 
& Smits, 2000). 

Both PBL and PrBL approaches promise to support the transfer of 
knowledge and skills from the classroom to the workplace. PBL 
and PrBL (Barrows, 1996; Marx, Blumenfeld, Krajcik, & Soloway, 
1997) focus on providing carefully selected or designed problems 
or challenges that require learners to apply domain-specific and 
domain-general knowledge, self-directed learning strategies and 
team participation skills. Whether solving a complex problem or 
using the design process to develop a new product while resolving 
arising issues, learners apply their cognitive skills to an increasingly 
varied set of real-world contexts and settings (van Merrienboer, 
Clark, de Croock, 2002). This would assist them in transferring 
their knowledge, skills and attitudes from the school setting to 
real world contexts (Clark, Kirschner & Sweller, 2012; Sweller, 
Kirschner & Clark, 2007). Recently, both approaches have become 
progressively more popular in higher education settings. In this 
article, we use PrBL as an overall curriculum focus and PBL as an 
embedded problem solving approach to achieve the goals of the final 
project or task. 

The Problem 

Despite the many appealing characteristics of PBL and PrBL, there 
are ongoing disputes about the degree to which students should be 
supported or guided with adequate feedback during the problem 
solving or product development process. On the one hand, research 
shows that both of these approaches are most effective when 
appropriate learning goals are defined; formative, continuous and 
effective feedback are provided; and multiple opportunities for self- 
assessment, peer-assessment and reflection are offered. Evidence 
further suggests that without explicit instructional guidance and 
feedback capable of enhancing the learning process (Nicol & 
MacFarlane-Dick, 2004), it is likely that the underlying concepts 
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and skills that students are required to learn will not be achieved. 
On the other hand, instructional support and formative feedback that 
can generate beneficial information for students could take many 
forms, and are still not well-articulated, particularly in web-based 
learning environments. A review of the literature shows that the 
effectiveness of feedback is reliant on several factors such as models 
of feedback, modes of feedback, forms of feedback, the timing of 
the feedback, and the ability and skills to give and receive effective 
feedback. However, how feedback should be implemented in PBL 
learning environments needs further exploration and articulation. 


The main purposes of this study were to (1) use the literature to 
build conceptual models that could guide instructional designers 
to integrate effective feedback and feed-forward in the design and 
development of an online graduate-level course utilizing problem/ 
project based learning; and (2) to assess the utility and impact of the 
models on student learning and satisfaction. 


LITERATURE REVIEW 

An extensive review of the literature was conducted, and the results 
were synthesized to develop two conceptual models for the above 
purpose. The following sections summarize the areas of review. 

Feedback 

Feedback is an essential component of learning and understanding 
which is commonly thought of as the information given to 
students about the quality of performance (Cohen, 1985; Sadler, 
1989). It often fulfills both summative and formative purposes 
and is delivered both through written text and verbal commentary 
(Price, 1997). The difference between summative and formative 
feedback is in their purpose and effect (Sadler, 1989). The purpose 
of summative feedback is to provide information on how well a 
student has achieved particular major learning targets (Nitko, 2004). 
These learning targets are typically those that are to be mastered at 
the end of a unit of instruction and can only inform for subsequent 
learning. Summative feedback generally includes a grade or mark 
as well as comments that provide insights on why the particular 
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mark is given. The primary purpose of formative feedback, on the 
other hand, is to allow the student to look at their current pieces of 
work, make adjustments to that work, and to carry those strategies 
to future work. Formative feedback is continuous and involves 
gathering information about students’ progress throughout a unit of 
instruction (Nitko, 2004) with the intention of improving learning 
(Sadler, 1998). To be effective, formative feedback should provide 
information about the gap between actual performance and some 
established standard of performance (Johnson & Johnson, 1993). 

Formative Feedback, Feed-up and Feed-forward 

Formative feedback can signal a gap between a current level 
of performance and some desired level of performance or goal. 
Resolving this gap can motivate higher levels of effort (Song & 
Keller, 2001). That is, formative feedback can reduce uncertainty 
about how well (or poorly) the student is performing a task (Ashford, 
1986; Ashford, Blatt, & VandeWalle, 2003). According to Sadler 
(2010), “if feedback is to have a reasonable prospect of achieving 
its formative purpose, it has to be both specific (referring, as it 
necessarily does, to the work just appraised) and general (identifying 
a broader principle that could be applied to later works)” (p. 3). 

Feed-forward provides guidance before students attempt their 
assessment task (Hendry, White & Herbert, 2016). By definition 
feed-forward is timely and future-oriented in relation to the upcoming 
task. It includes giving students ‘tips’ or guidance about what not 
to do based on previous performance and involving students in 
judging the quality of their work (Hattie & Timperley, 2007; Nicol 
& Macfarlane-Dick, 2006). 

Hattie and Timperley (2007) argue that feedback and feed-forward 
strategies can be first applied by answering the following three 
questions to offer a feed-up. 

1. Where am I going? What are the goals and how can I attain 
them? 

2. How am I doing? Am I meeting the said goals and how is my 
current performance? and 

3. Where to next? What can I do to prepare or better understand 
for future tasks? 
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These three questions assure that the feedback helps explain what 
quality performance is (feed-up), provides opportunities to close the 
gap between current and desired performance (feedback ), and helps 
the student see the next steps in learning and how to undertake them 
( feed-forward ) (Hattie & Timperley, 2007; Nicol & Macfarlane- 
Dick, 2006). This strategy for developing comprehensive feedback, 
however, may not be enough to close the gap. Students must be 
able to act upon this feedback to make changes to their own work. 
Problems may arise if students have difficulty understanding the 
comments or knowing how to act upon the suggestions (Carless, 
Salter, Yang, & Lam 2011). Feedback then cannot be only about 
telling and dispensing information to the learner (Sadler, 2013). For 
feedback and feed-forward to be successful, a set of characteristics 
must be present in order to close the gap between the learners current 
and desired performance. 

Feedback, feed-up and feed-forward can be provided in various forms 
(written, oral, dialogical), offered immediately/delayed, frequently or 
in a timely manner, serve different functions (directive, facilitative, 
specificity) and come from a variety of sources (instructor’s one-on- 
one interaction, instructor’s small group interaction, peers and self). 
See Appendix A for the synthesis of the literature in each of these 
areas. 

Problem-Based and Project-Based Learning 

Both PBL and PrBL involve a general process and a set of 
characteristics that ensure the learning experience provided to 
the learner is student-centred (Barrows, 1996) and that learners 
are engaged in authentic problems that will encourage learning, 
metacognition, and self-regulation (Torp & Sage, 2002). Table 1 
summarizes the characteristics and the process of implementing 
PBL or PrBL. 

The Instructor’s Role in PBL 

The facilitator’s/instructor’s role in PBL is critical to successful 
implementation and requires careful facilitation of the learning rather 
than just delivering the content (Savery, 2009). This facilitative 
role requires instructors to develop learning goals with the students 
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(Markham, Larmer & Ravitz, 2003), monitor the learning process 
to ensure that no phase of the learning process is being overlooked, 
and encourage the students’ active participation (Barrows, 1988). 
The instructor/facilitator must constantly probe the students with 
questions to challenge their understanding of the problem and guide 
them to the deeper levels of understanding (Savery & Duffy, 1995; 
Barrows, 1988). Another significant role for instructors is to provide 
frequent milestones for students to measure success and afford 
opportunities for formative assessments and feedback throughout 
the learning experience to monitor progress (Markham, Larmer, & 
Ravitz, 2003; Barrows, 1988). Barrows (1988) suggests that “focused 
metacognitive questions” (p. 10) can be used to probe the students in 
the areas where they may be having difficulty, therefore increasing 
the students’ ability to notice these inadequacies in future learning. 
Finally, the instructor must continuously provide opportunities for 
students to discuss, reflect and evaluate their learning (Markham, 
Larmer & Ravitz, 2003). 


Table 1 


Characteristics and process of implemen ting PBL or PrBL 


Characteristics 


Process of Implementation 


• Problems/Tasks must be 
authentic, open-ended, aligned 
with curriculum standards, 
and must integrate multiple 
disciplines. 


• The students define the problem/ 
task, and a clear set of learning 
goals is developed. 


• The problem should engage • Students become actively 
students in a comfortable engaged with the problem 

cognitive conflict or dissonance as stakeholders and generate 
as well and should utilize vital possible ideas and solutions 
workplace skills and enhance using present levels of 

student self-regulatory skills. knowledge. 


• The instructor facilitates, • Students must conduct research 

coaches, and or supports the or inquiry to fill in the gaps and 

development of the students’ acquire the knowledge and skills 

metacognitive processes to necessary to solve the problem, 

develop the “problem-solver.” 


(continued) 
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Characteristics 

• Requires the application of both 
domain-specific and domain 
general knowledge to solve or 
develop the product. 


• Opportunities for collaborative 
learning environments should be 
provided. 


• Promotes 21 st Century learning 
skills such as communication, 
collaboration, creativity, and 
critical thinking. 


Process of Implementation 

• Students continue to generate 
an exhaustive list of solutions 
and choose the best based on 
close analysis of data collected 
during research and inquiry. The 
solution can be presented as a 
product, performance, or event. 

• Authentic assessments are 
used throughout the learning 
process allowing students to 
demonstrate their learning. 

• Frequent debriefing activities 
(group meetings, journaling, 
etc.) take place to ensure 
reflection is occurring and key 
concepts are being learned. 
(Barrows, 1988; Barrows, 1996; 
Larmer & Mergendoller, 2013; 
Savery, 2009; Savery & Duffy, 
1995; Torp& Sage, 2002), 


PrBL or PBL in Online Environments 

Online learning, also known as web-based instruction, occurs when 
instructional design strategies utilize the World Wide Web as a 
resource to integrate a variety of information and for collaboration 
and communication of ideas (Relan & Gillani, 1997). Distance 
education is often an added component to online learning and 
involves instruction that takes place when the learning group is 
separated by space and or time and where telecommunication 
systems are used to connect the learners, resources and instructors 
(Simonson, Smaldino, Albright, & Zvacek, 2015). Because PBL/ 
PrBL requires frequent collaboration, communication and various 
forms of information and resources, it is essential that the virtual 
learning environment provides a set of learning and teaching tools 
to support this. These tools should enhance the students’ learning 
experience rather than hinder it, and can include synchronous and 
asynchronous methods such as email, webcasts, discussion boards, 
learning management systems and teleconferencing (Savin-Baden 
& Wilkie, 2006). 
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There are often challenges to creating effective PBL instruction in 
an online environment. First, for instructors, it can be difficult to 
manage the resources and information necessary while also engaging 
students in a collaborative online process (Savin-Baden & Wilkie, 
2006). The separation of the instructor, peers, and the student can 
also make collaboration over an authentic problem difficult. Space 
and time limit the amount of interactions and the ability to meet as 
groups to effectively plan and manage group activities. A student’s 
motivation and responsibility for his/her own learning also presents 
a problem. Learners often already feel that a large amount of 
autonomy is necessary in PBL as well as in an online environment. 
When combined, students can feel overwhelmed and experience 
loss of control (Savin-Badin & Wilkie, 2006; Simonson, Albright 
& Zvacek, 2015). 


Feedback in PBL/PrBL: Design Models 

Analysis and synthesis of the literature provided the basis for 
constructing two conceptual models to guide instructional designers 
in integrating effective feedback, feed-up, and feed-forward in the 
design and development of an online graduate level course utilizing 
problem/project based learning. The proposed models presented 
in Figure 1 and Figure 2 originate from principles and ideas 
reviewed in the literature, but each focuses on different processes in 
designing instruction for PBL/PrBL. Figure 1 depicts a model that 
an instructional designer can use to shape a course to meet the needs 
of supportive feedback in PBL/PrBL environments by ensuring that 
all of the necessary characteristics of feedback are present. Figure 2 
provides a model that shows the process used to implement a feed- 
up, feedback and feed-forward loop to ensure continuous learning 
and performance improvement in a PBL environment. 

The model in Figure 1 uses the principles of PBL/PrBL to develop a 
set of characteristics for feedback necessary to support achievement 
and ongoing learning in PBL. The implementation of these 
characteristics is further clarified with suggestions on how designers 
can apply them. 
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Figure 1. Characteristics and strategies for feedback in PBL 
design model 


Figure 2 depicts a model of the feedback loop that can be used to 
foster student learning in a PBL environment. This loop employs 
the concepts of feed-up, feedback and feed-forward to provide 
ongoing support, facilitation and coaching by the instructor, peers, 
and learners themselves. Feed up in this model is defined as the 
information established about the attainment of the learning goals. 
This involves providing clear success criteria and setting goals for 
ongoing learning within the course or PBL/PrBL process. It must 
be noted that the success criteria and goal setting do not have to be 
provided by the instructor and that collaborative student development 
of these items is encouraged in PBL. Feedback is defined as the 
information provided about performance in relation to the criteria or 
goals. This written or verbal feedback can be done by the instructor 
or coach, but can also involve peer, small group, and self-assessment 
feedback strategies. Feed-forward is defined as information given to 
learners that leads to future goal attainment. Although not required, 
feed-forward strategies can best be employed when engaging in 
dialogue between an instructor and the learner. This allows the 
learner to help build these learning goals with the instructor, ask 
clarifying questions, and discuss how they will apply them to future 
works. Through this dialogue and the setting of goals for future 
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learning, students will develop and improve on metacognitive 
skills such as planning, evaluating, and making adjustments. It is 
important to note that this feedback loop is continuous throughout 
the PBL process and does not start or end at any clear point. 


Feed up 

Provide clear success 
criteria 

Set goals for ongoing 
learning 

Create collaborative 
teams 



PBL Task/ 
Work Product 


Feed-forward 

Feedback 

• Information about 

• Information that leads 


performance in 

the learner toward 


relation to criteria or 

future goal 


goals 

attainment. 


• Instructor/coach 

• Dialogic 


• Peer/Small Group/Self 

• Metacognitive 


• Written & Verbal 


Figure 2. The Feedback loop in PBL design model 


Applying the Proposed Models to an Online Graduate Course 
Course Description 

An online graduate-level course was selected to test the application 
of the proposed models. The course is entitled “Organization and 
Management of Instructional Technology” and is designed using 
the PBL model as its framework. It is an elective course offered 
to students in an instructional technology graduate programme. 
Participants enrolled in this course are primarily graduate students 
seeking a master’s degree or a certificate in instructional technology, 
particularly those who have chosen public and private education as 
their programme track. The course closely examines the planning 
and management of a technology change in public or private schools 
as well as other organizations. Applying the PBL approach, students 
are engaged in a series of authentic problem-solving tasks and 
collaborative team activities that lead up to a culminating project. 
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The culminating project is divided into three main sections that are 
due at various points throughout the semester and requires students 
to build on previous sections to form the final culminating product. 

The course is delivered using Blackboard learning management 
system (LMS) along with WebEx synchronous learning tools to 
provide both synchronous and asynchronous learning opportunities 
to students at a distance as well as in a face-to-face classroom. 
WebEx allows classes to hold a whole group discussion as well as 
breakout rooms for team meeting areas. A total of 11 students were 
enrolled in the course during the implementation of this study. The 
designer of the course was also the instructor for the course delivery. 

Course Design Features 

As previously mentioned, this course was designed using the PBL 
instructional design method to allow learners to apply skills and 
knowledge to the practical application of an actual technology 
change project. To support the PBL process, specific characteristics 
and strategies for feedback were also implemented throughout the 
course as conceptualized in the Figure 1 model. These included 
strategies that provided evidence of applying learner-centred 
approach, promoting opportunities for critical thinking skills, and 
providing authentic application of content and skills (see Table 2). 


Table 2 


Application of the Feedback Model in a Graduate Level On line PBL 
Course 


PBL 

Characteristic 


Feedback 

Characteristic 


Strategies Applied in Online 
Graduate Course 


Learner Centred Timely 

Feedback 
must attribute 
students’ success 
and promote 
independence. 


• Larger project is broken down into 
smaller parts and learning modules with 
complimentary activities. 


• Feedback is given before another 
activity begins. 


Frequent 


• Weekly feedback is provided 
throughout the course that builds on 
each other for the final product. 


(continued) 
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PBL 

Characteristic 


Promotes 21 s1 
Century Skills 

Feedback must 
use a variety 
of modes and 
methods. 


Feedback Strategies Applied in Online 

Characteristic Graduate Course 


Motivating 


Feed-Forward 

Goals 


Small Group 
and Peer 


Self-Assess and 
Reflect 


Verbal and 
Written 


Dialogic 


• Attribute students’ success to effort. 

o “I think your model is appropriate 
because... ” 

o “You did a good job of explaining 
how you will... ” 

• Offer information about strategies or 
processes to improve on future tasks or 
set goals. 

o “. . . you need to make sure you are 
putting emphasis on the outcome 
and not so much the process" 

o “[tjhink about it and make sure 
you improve your argument when 
writing [the] final tech plan. ” 

• Dialogue in which peers are encouraged 
to openly make suggestions about other 
teams’ work. 

• Discussion board posts in Black Board 
LMS where students can provide peer 
feedback. 

• Students use success criteria provided to 
measure how they feel they performed 
on a given task. 

• Reflection and self-assessment is 
encouraged but not assigned. 

• Reflective dialogue at the beginning 
of each synchronous class reported by 
each team to the whole group. 

• Verbal feedback is given weekly via 
whole group discussion and one or 
more times during team meetings. 

• Written feedback is provided for all 
three project submissions and many 
practice activities. 

• Discussions during synchronous classes 
among peers and the instructor on how 
the projects were progressing, areas for 
improvements, and successes. 


(continued) 
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PBL 

Characteristic 

Feedback 

Characteristic 

Strategies Applied in Online 
Graduate Course 

Authentic 

Informative 

• Feedback is given which supports the 

Application of 

and Specific 

learning of content. 

Content & Skills 


o “You explain your change model, 
but do not explain how you would 
use the model to assist you in 
planning and implementing the 
change. ” 

o “You might need to think about the 
sequence of your training, and the 
staff might need a more hands on 
approach. ” 


Metacognitive 

• Feedback is given in the form of 
questions which guides the learner to 
elaborate or clarify ideas, 
o “ What should you expect after one 
year?” 

o “ What is the outcome goal you are 
trying to achieve?” 


The Impact of the Feedback Models on Student Performance 
and Satisfaction 

Research Questions 

The purpose of this study was to use the literature to build a 
conceptual model to integrate effective feedback, feed-up and feed¬ 
forward in the design, development and implementation of online 
courses, and then test the model in an online graduate level course 
utilizing PBL. The latter part of the study was specifically focused 
on the following questions: 

1. How was the course addressing feedback, feed-up, and feed¬ 
forward characteristics and strategies to support PBL? 

2. How were students utilizing feedback opportunities? 

3. What were students’ perceptions about the feedback and feed¬ 
forward provided in the course? 

4. How did feedback and feed-forward strategies influence 
student performance and satisfaction? 
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Participants 

The results presented in this study are based on both quantitative 
and qualitative analysis of data gathered from 11 graduate level 
students (10 females and 1 male) enrolled in one semester of a web- 
based course. The majority of the students worked full time during 
the day and were part-time graduate students with job experience 
in education, business, journalism, marketing, programming, and 
health care. The students’ age ranged from 29 to 50, with 62% 
with a college degree and 37% with a post-college degree. For the 
course, students worked in teams of two with the exception of one 
student working on the project alone. There were six teams in total for 
this course. 

Data Collection Methods 

Data was collected using a variety of methods over the course of the 
semester. See Table 3 for the data-gathering strategies. 

Table 3 

Summary of Data Collection Methods 


Questions of the Study 

1. How was the 
course addressing 
feedback, feed-up 
and feed-forward 
characteristics and 
strategies to support 
PBL? 

2. How were students 
utilizing feedback 
opportunities? 


3. What were students’ 
perceptions about the 
feedback and feed¬ 
forward provided in 
the course? 


Data Collection Methods 

• Synchronous Class Observation: Observations 
of live course discussions were conducted to take 
notes about student dialogue, instructor’s verbal 
feedback, and peer feedback. 

• Class/Team Discussion Boards: Student postings 
on weekly assignments in these discussion 
boards were analyzed for instructor feedback, 
peer feedback, and student reflection where 
applicable. 

• Personal Blackboard Areas: Student postings 
were used to analyze reflections and self- 
assessments where applicable. 

• Student Performance Results: Student assignments 
and projects were used to determine feedback 
characteristics used by the instructor. Data was 
also collected on how students performed on each 
assignment as well as how often feedback was given. 


(continued) 
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Questions of the Study 

Data Collection Methods 

4. How did feedback 
and feed-forward 
strategies influence 
student performance 
and satisfaction? 

• Student Survey: An online survey was given to 
students near the end of the semester. This survey 
included questions about student demographics, 
the perception of the feedback provided in the 
course, and the overall satisfaction with the 
feedback used. Satisfaction was rated on a 
5-point scale as follows: 0=N/A, l=Strongly 
Disagree, 2=Disagree, 3=Neutral, 4=Agree, 
5=Strongly Agree 


Qualitative data was gathered using notes from synchronous 
class observations, entries from class and team discussion boards, 
instructor feedback on assignments, and open ended survey 
questions. This data was examined for student/instructor dialogic 
feedback, and instructor’s verbal and written feedback. During the 
live synchronous class observations, notes were taken on what the 
students and instructor said during instances of feedback. If the 
notes were not clear enough to get the direct quotes due to speed of 
conversation, course recordings were used to clarify. 

Analysis 

Quantitative data was analyzed using SPSS software application. 
Qualitative data was coded using the following rubric (see Appendix 
B for examples). 

• Motivating: If the comment attributed the students’ success 
to effort. 

• Feed-Forward: If the comment offered information about 
strategies or processes to improve on future tasks or set goals 

• Informative/Speeific: If the feedback directly supported the 
learning of the content 

• Metacognitive: If the feedback was given in the form of a 
question to guide the learner to elaborate or clarify. 

Using the above list of feedback characteristics, each of the feedback 
comments (phrases; sentences) made by the instructor was read 
and coded. The comments were analyzed for their characteristics, 
and each segment of the comment was assigned to the appropriate 
category. Once complete, a comparison was made between the total 
number of feedback comments and the total number of comments 
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in each category to calculate a percentage. It should be noted that 
the percentages were only based on the information gathered during 
small class segments over the semester, totaling roughly 30 minutes. 
Long feedback segments of dialogue with multiple statements were 
counted as one feedback comment unless the feedback shifted from 
one characteristic to another. For example, if the instructor spoke 
for three minutes, giving informative/specific feedback to clarify a 
point, and then ended with a metacognitive question, these counted 
as two comments (one for each category). 

The same method was used to code the class team and discussion 
boards, the instructor feedback on assignments (see Appendix B for 
examples), and the open-ended survey questions. Comments were 
read, analyzed, and categorized based on feedback characteristics. 
The totals were counted. The data was coded by one of the 
researchers. The two researchers then met, reviewed portions of the 
coding for data sources and if there was disagreement, discussed the 
coded segments of the data and coded them again. 


Findings 

How was the course addressing feedback, feed-up and feed¬ 
forward characteristics and strategies to support PBL? Tables 
4 and 5 show the frequency and/or method with which various 
feedback (also see Figure 1) were addressed throughout the semester 
of the graduate level online PBL course. Table 4 summarizes the 
characteristics of feedback present when written forms are used. 
To gather this data, the researchers looked at the written feedback 
on all papers submitted (both final and draft forms) to determine 
the timing and frequency of motivating, feed-forward, informative 
or specific, and metacognitive feedback. The results show that 
feedback was timely, with all reviews being delivered to students 
within 10 days or less of project or draft submission. Feedback 
frequency was dependent upon student request (discussed further 
in the next section), but at least two extensive forms of written 
feedback were given to students up to the point of data collection. 
As shown in Table 4, the majority of the written feedback for this 
course was informative/specific or motivating. Feed-forward and 
metacognitive feedback was given less often, although not by a 
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large percentage. It is also evident that feed-forward comments 
occurred more frequently during the review of student drafts rather 
than on final projects. This is probably because drafts are used to 
improve the performance of a final product, and feed-forward goals 
are effective at achieving this goal. 


Table 4 

Written Feedback Characteristics in the Course (see examples in 
Appendix B) 


Occurrence 


Timely 

• 

Project draft feedback given within 0-7 days of stu¬ 
dent request. 


• 

Final project feedback received within 10 days of 
student submission. 

Frequent 

• 

Feedback given upon student request with project 
drafts. 


• 

Given 2 times (once for each project) for final proj- 



ects. 

Motivating 

• 

25% of comments on Project 1 Final 


• 

32% of comments on Project 2 Final 


• 

23% of comments on Drafts. 

Feed-Forward Goals 

• 

13% of comments on Project 1 Final 


• 

13% of comments on Project 2 Final 


• 

23% of comments on Drafts. 

Informative/Specific 

• 

48% of comments on Project 1 Final 


• 

35% of comments on Project 2 Final 


• 

34% of comments on Drafts. 

Metacognitive 

• 

14% of comments on Project 1 Final 


• 

20% of comments on Project 2 Final 


• 

19% of comments on Drafts. 


Table 5 shows the characteristics of feedback applied during verbal 
or dialogic portions of the course. To gather this data, observations 
were conducted during dialogic feedback portions of the whole group 
live asynchronous classes. These small segments of various classes 
totaled approximately 30 minutes. According to the data in Table 5, 
the instructor utilized metacognitive questioning and informative/ 
specific feedback to students most often during verbal feedback 
dialogue. It is possible that the frequent use of these methods was 
because conversation lends itself to metacognitive questioning as 
well as clarification of information that is not understood. Feed- 
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forward goals and motivating comments occurred less often, but 
were a small part of the dialogic conversation. 

Table 5 

Verbal/Dialogic Feedback Characteristics in the Course 


Occurrence 

Timely 

• 

Weekly dialogic feedback through class discussions 
immediately following class assignments and project 
work. 

Frequent 

• 

Weekly dialogic feedback given through class 
discussions. 

Motivating 

• 

14% of comments were motivating. 

Feed-Forward Goals 

• 

14% of comments provided feed-forward goals. 

Informative/Specific 

• 

35% of comments were informative or specific. 

Metacognitive 

• 

37% of comments were metacognitive questions. 


Note: Percentages are based on a sampling of class segments totaling 25 minutes of 
dialogic whole group feedback. 


It is evident that the characteristics of feedback necessary to support 
PBL were addressed in this course. Data from Tables 4 and 5 show 
that motivating feedback was given most often in written form, 
whereas metacognitive feedback occurred most often during verbal 
dialogue. Informative or specific feedback was given frequently in 
this course, perhaps due to the complexity of the tasks at hand and 
the lack of prior knowledge among students in the class. Timeliness 
and frequency of the feedback most often depended on the students’ 
utilization of feedback opportunities provided in the course. The 
following section explores this area more closely. 

How were students utilizing feedback opportunities? As feedback 
can vary depending on student needs and their motivation to seek out 
feedback opportunities, it is critical to look at how students chose 
to use the feedback opportunities in this course. The Blackboard 
learning management system was used to determine how often 
groups were posting drafts, activities, and comments in the group 
discussion and team discussion areas for instructor feedback, peer 
feedback, and self-assessments. Emails to the instructor were also 
calculated. Because all work products were done as teams in the 
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course, the results show team utilization of Blackboard rather than 
that of individual students. Table 6 highlights the frequency with 
which teams utilized these feedback opportunities. 

Table 6 

Students Utilizing Feedback Opportunities 


Team A Team B Team C Team D Team E Team F 


Team Communication 

0 

4 

0 

2 

0 

0 

Reflective Posts 

1 

1 

1 

3 

2 

2 

Questions Raised 

0 

1 

0 

0 

0 

2 

Feedback to Peers 

2 

0 

0 

1 

1 

1 

Project Drafts and Finals 

9 

3 

2 

3 

3 

3 

Other Posts 

10 

12 

9 

10 

10 

16 

Total Posts 

22 

21 

12 

19 

16 

24 

Instructor Feedback 

6 

4 

2 

4 

2 

11 


Note: The numbers note the frequency with which the team utilized each feedback 
opportunity. 


As evident in Table 6, few teams utilized the Blackboard LMS to 
communicate as a team, ask questions, and provide feedback to peers. 
These practices only made up 10% of the total usage altogether. 
Some teams did use email to communicate with the instructor 
instead of through Blackboard, and there was no way to know how 
often peers communicated with each other through email or other 
communication tools such as Skype, Google Drive, Dropbox. The 
majority of posts (78.9%) were project drafts, final drafts, and class 
assignments. Reflective posts made up 8.8% of the overall posts. It 
was noted that the more frequently a team posted in Blackboard in 
total, the more often they received feedback from the instructor. 

What were students’ perceptions about the feedback and 
feed-forward provided in the course? Data was collected using 
a survey to determine students’ perception about how formative 
feedback was delivered, the effectiveness of each mode and method, 
as well as how students felt they utilized the feedback and course 
materials. Table 7 indicates the level of satisfaction students had 
with how the instructor provided formative feedback throughout 
the course. It was found that students agreed overall that formative 
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feedback was effective in this online graduate level PBL course, 
with a fairly low standard deviation. Further analysis showed that 
the most highly rated characteristics (M=4.2500) were feedback 
that (a) clarified good performance (goals, criteria, standards); 
(b) provided opportunities to repeat the same task performance 
feedback cycle by allowing re-doing; and (c) was relevant to the 
task at hand and their needs. The lowest ratings (M=3.750) was 
given to feedback that clarified task requirements and made criteria 
for complex tasks explicit. One student surveyed disagreed that 
feedback was descriptive both in writing and oral modes. However, 
other classmates agreed or strongly agreed with this statement. It 
might be concluded that students felt the performance criteria was 
clear and relevant (feed-up), but could have been more specific to 
support complex tasks. 

Table 7 

Student Perception of Formative Feedback 


Statement 

M 

SD 

Various forms of feedback assisted me in reflecting and self- 
assessing my learning. 

4.125 

.35 

The instructor’s formative feedback helped clarify good performance 
(goals, criteria, standards). 

4.250 

.46 

The formative feedback on the assignments, quizzes, and project 
parts provided delivered high quality information about students’ 
learning. 

4.125 

.64 

Feedback clarified task requirements. 

3.750 

.71 

Feedback was descriptive both in writing and oral. 

3.875 

.83 

Feedback made criteria for complex tasks explicit. 

3.750 

.46 

Feedback provided opportunities to repeat the same task performance 
feedback cycle by allowing re-doing. 

4.250 

.71 

Feedback was relevant to the task at hand and my needs. 

4.250 

.46 

Feedback assisted me to know what is expected and how to get there. 

3.875 

.35 


N=8 

5 Point Scale: 0=N/A, l=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 
5=Strongly Agree 


Further analysis was conducted on the students’ perceptions on their 
use of course materials and instructor feedback (see Table 8). This 
data indicated slightly more variance (SD >.53 on all items) among 
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students, and it was more difficult to draw conclusions. It may be 
important to note, however, that a majority of students selected were 
either neutral or disagreed that they “did not know what to expect, 
thus I sought the instructor’s feedback.” It is unclear whether they 
felt they knew what was expected, or if they felt they did not seek 
out the instructor’s help. Perhaps this was the reasoning for so many 
neutral responses. 

Table 8 

Student use of Course Materials and Feedback 


Statement 

M 

SD 

I constructed a personal interpretation of the requirements and 
properties of the tasks that I had to complete. 

3.875 

.64 

I formulated my own task goals and engaged in actions to achieve 
these goals by applying tactics and strategies of my own. 

4.125 

.64 

I monitored my performance and used my own self-assessment to 
generate feedback to improve. 

4.000 

.53 

I did not know what to expect, thus I sought instructor’s feedback. 

3.250 

.71 

N=8 




5 Point Scale: 0=N/A, l=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 
5=Strongly Agree 

Overall, students were satisfied with the modes and methods of 
delivering feedback in this course (see Table 9). Students were most 
satisfied with written comments on assignments and other work 
products (M = 4.375) as well as the effectiveness of examples and 
sample projects (M = 4.250). Least effective were the automated 
electronic forms of feedback. Two possible reasons for this could 
be that there was only one opportunity in the course for electronic 
automated feedback, or that students were not aware of how to 
properly receive this feedback after taking the test. 

Of all the survey items, the most telling were the students’ comments 
on the open-ended questions (Table 10). When asked overall, how 
they evaluated the quality and appropriateness of the feedback in 
this course, almost all the students surveyed stated that they felt 
it was effective, of good quality, and essential to their success 
on assignments. Only one student gave a suggestion pertaining 
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to the organization of the learning management system. This led 
to the second open-ended question in this survey that asked what 
suggestions the students had for possible feedback strategies. 
Many students indicated that a reorganization of Blackboard 
was necessary to help find assignments, feedback, and rubrics. 
Comments indicated that it was difficult to search the many places 
where items were located, and that they were not aware of when and 
where to find assignments and rubrics. Additionally, students stated 
that clarification of expectations (feed-up) could be improved by 
providing more examples of quality performance or increasing the 
number of one-on-one instructor meetings. Based on the comments, 
it is possible to assume that students were satisfied with the feedback 
they received but wished that feed-up information was easier to find 
and use to improve performance before feedback was given. 

Table 9 


Student Perception of Feedback Modes and Methods 


Statement 

M 

SD 

One-on-one or team meeting was effective. 

4.000 

.53 

Electronic automated feedback was effective. 

3.875 

.64 

Class discussion and instruction was effective. 

4.000 

.53 

General explanation and clarification of criteria was effective. 

4.000 

.53 

Examples and sample projects and responses were effective. 

4.250 

.71 

Break down of the tasks/requirements was effective. 

4.125 

.64 

Written comments on assignments and other work products were 
effective. 

4.375 

.74 


N=8 

5 Point Scale: 0=N/A, l=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 
5=Strongly Agree 


How did feedback and feed-forward strategies influence student 
performance? According to the literature, feedback and feed¬ 
forward strategies are necessary for students to be successful in a 
PBL course, especially in an online environment. It is often noted, 
however, that motivation and self-regulation are necessary for 
students to benefit from feedback and to become successful in the 
types of critical thinking activities needed in this course. Due to this 
element, data collected on the types and frequency of team feedback 
in relation to student performance were compared (see Tables 11 and 
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12). Table 11 highlights how effectively students utilized feedback 
opportunities to improve performance, while Table 12 focuses on 
the instructor’s implementation of feedback to help teams improve. 
Based on the data, it is difficult to claim that the frequency or type 
of feedback helped students who received high grades on projects. 
It is however evident that high-achieving teams did receive more 
motivating feedback. This is likely due to the fact that the comments 
about good performance were in direct relation to high scores on 
the rubric, which meant that the instructor let teams know what they 
were doing well, while also scoring them high on the rubric. 

Table 10 

Student Comments about Feedback in the Course 


Assigned Codes 
Effectiveness of 
Feedback 


Design of Course 
LMS 


Student Comments 

Overall, the quality and appropriateness of the feedback 
was effective. 

Feedback on projects was usually useful information 
that helped me make changes and understand what I 
could have done a little better. It was appropriate to the 
task at hand. 

The feedback has been crucial to my success in this 
course. Every bit of feedback both noticed and unnoticed 
has helped me to gain a better understanding of the 
course materials and processes. All feedback was given 
in appropriate forms, quality, and timeliness. 

Very effective to get feedback throughout the course. 
The quality was good and challenged to be better. 

The only challenge I found was sometimes it was 
difficult to navigate the Blackboard and find where tasks/ 
assignments were posted. There would be assignments 
posted that I didn’t know about. 

It was a little difficult finding some of the rubrics. I 
spend a lot of time just sifting through the website. If 
you could organize the blackboard page a little better 
and check links before that week, it would help students 
that need to work ahead some weeks rather than falling 
behind on busier weeks. 

There are several varieties of feedback and places to find 
information throughout the course, but sometimes it can 
be overwhelming to have so many different places to 
look or check. 


(continued) 
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Assigned Codes 
Clarification of 
Expectations 


Other Suggestions 
for Feedback 


Student Comments 

Sometimes I would like more concrete examples of what 
would constitute an above average answer in the written 
feedback. 

More examples presented. 

More clarification on expectations. 

More instructor meetings for each step. 

Also, having the rubrics at the beginning of a project 
is more helpful than in the middle or towards the end, 
because I ended up going back and adding things after 
finding the rubric. 

A potential for this class and many others in the MIT 
program would be team skill feedback from the 
professor (as applicable and possible) and from team 
partners. This feedback would definitely best to happen 
at the end of the course but perhaps also midway through 
to encourage positive changes that can be observed at the 
end of the course. 

Less discussion and more in class activities. 


What is most evident when comparing the data in Tables 11 and 12 
is the effect that posting in Blackboard had on overall performance. 
The two teams that posted the fewest amount of times in Blackboard 
(only 12 and 16 times) also showed the lowest performance on one 
or both of the projects (70, 88, 89). Students who were not seeking 
out as much instructor support struggled more in the beginning. 
These two teams, however, did receive the most informative/ 
specific feedback (53.1% and 48.2%) and relatively high rates of 
metacognitive feedback (15.6% and 17.2%). Team E was able 
to use this highly effective feedback provided to improve critical 
thinking about the task and to improve both the scores from Projects 
I and II. Team C’s score did not improve, but this could have been 
because this team consisted of only one person. As PBL is highly 
collaborative in nature, Team C may have had difficulty thinking 
through the complex problem solving tasks alone. 

It is also important to note that in the end, all teams scored relatively 
well on both projects (85-99%). This finding suggests that the 
levels of motivating-feedback, feed-up and feed forward goals, 
informative/specific feedback, and metacognitive feedback were 
adequate to support student achievement. Team B did not seek as 
much help from the instructor via posting drafts and assignments, 
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but was still successful. This might have been due to the team’s 
overall understanding of the material and their confidence that they 
were moving in the right direction. 

Table 11 

Team Feedback and Student Performance 



Team 

Team 

Team 

Team 

Team 

Team 


A 

B 

C 

D 

E 

F 

Projects/Drafts Submitted 
(Frequency) 

3 

2 

2 

4 

3 

5 

Motivating Feedback (%) 

14.8 

58.3 

21.9 

22.9 

31 

29.2 

Feed-forward Goals (%) 

18.5 

12.4 

9.4 

20.8 

3.4 

15.4 

Informative/Specific Feedback 
(%) 

42.6 

16.7 

53.1 

47.9 

48.2 

36.9 

Metacognitive Feedback (%) 

24 

12.5 

15.6 

8.3 

17.2 

23.1 

Project I Grade (%) 

90 

93 

89 

93 

70/93 

93 

Project II Grade (%) 

85 

99 

88 

98 

99 

98 


Note: The numbers in row 1 indicate the frequency of students’ submission of 
projects or drafts to be reviewed. The numbers in rows 2-5 indicate the percentage 
of each type of feedback received by the team. The last 2 rows are the grades teams 
received on final projects. Team E was afforded two attempts at Project I but both 
scores are indicated to show team improvement. 


Table 12 

Team Feedback and Student Performance 



Team 

A 

Team 

B 

Team 

C 

Team 

D 

Team 

E 

Team 

F 

Projects/Drafts Submitted 

3 

2 

2 

4 

3 

5 

Motivating Feedback (%) 

14.8 

58.3 

21.9 

22.9 

31 

29.2 

Feed-forward Goals (%) 

18.5 

12.4 

9.4 

20.8 

3.4 

15.4 

Informative/Specific Feedback (%) 

42.6 

16.7 

53.1 

47.9 

48.2 

36.9 

Metacognitive Feedback (%) 

24 

12.5 

15.6 

8.3 

17.2 

23.1 

Project I Grade (%) 

90 

93 

89 

93 

70/93 

93 

Project II Grade (%) 

85 

99 

88 

98 

99 

98 


Note: The numbers in row 1 indicate the frequency of students’ submission of 
projects or drafts to be reviewed. The numbers in rows 2-5 indicate the percentage 
of each type of feedback received by the team. The last 2 rows are the grades teams 
received on final projects. Team E was afforded two attempts at Project I but both 
scores are indicated to show team improvement. 
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DISCUSSION AND CONCLUSIONS 

A literature review on feedback and PBL aided in designing and 
developing two design models that were then implemented in an 
online graduate course. The characteristics and strategies suggested 
in the models shown in Figure 1 and 2 have proven to support student 
achievement on course projects and activities. All students in the 
course were able to complete Projects I and II with an average or 
above average performance score. The timely and frequent feedback 
allowed students to make necessary changes to project parts and 
assignments to improve performance, and students commented that 
this helped them achieve success in this course. As the frequency 
of feedback was dependent upon student request, those that needed 
more frequent feedback during the acquisition of learning (Salmoni, 
Schmidt, & Walter, 1984) could benefit from it, but those did not 
need frequent feedback, such as Team B, could receive the moderate 
amount necessary to be successful (Lam et al., 2011). 

The motivating, informative/specific feed-up, feedback and feed¬ 
forward goals provided a reference for areas of student improvement. 
Comments in class indicated that students were “using the feed-up 
and feedback to make changes” and thankful “for such specific 
feedback.” A balance of these three types of feedback along with 
metacognitive questioning supported student improvement. Specific 
and informative feedback provided direct information that students 
could use to shape a foundation for understanding of the content 
(Goodman, Wood, & Hendrickx, 2004). The feed-forward goals 
helped students see the next steps needed for learning, and how their 
previous efforts could be changed to make improvements (Hattie & 
Timperely, 2007; Nicol & Macfarlane-Dick, 2006). Metacognitive 
questioning allowed learning to shift from the teacher to the learner 
(Salder, 1989) and forced students to think of ways to answer those 
questions to improve their work (Barrows, 1988). 

Although small group and peer feedback were delivered in this 
course mainly through whole group class discussions, often students 
learned from other’s processes during this dialogue. Hearing the 
mistakes, successes, and suggestions of other groups allowed 
teams to take in this information and then apply it to areas of their 
own work. Self-assessment opportunities, although not a direct 
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requirement for this course, were afforded to students. Some teams 
utilized this opportunity through Blackboard more often than others. 
However, there was no clear way to assess whether or not teams 
utilized this strategy other than through the survey responses. It is 
recommended that in future, students could be required to do a team 
self-assessment on each project draft submitted to allow them to 
think through their performance and find their own strengths and 
weaknesses first. 

Blackboard played a critical part in the success of students in this 
course, but also posed some concerns. PBL in an online environment 
requires careful attention to the learning and teaching tools in 
this virtual learning space (Savin-Baden, & Wilkie, 2006). The 
instructor provided various forms of feed up, feedback, and feed¬ 
forward strategies within this course management system. Students, 
however, commented that these learning tools were difficult to 
locate, or did not seem to work when needed. Findings suggest that 
it may be necessary for students to be offered instructions at the 
beginning of the course on exactly where all information would be 
located, where they are to post, and how they can communicate with 
each other and the instructor. This information should be clear and 
straightforward, perhaps with only one route to finding it. Keeping 
this step simple will allow students to focus on the difficult tasks and 
critical thinking necessary for PBL. Instructors may also consider 
ensuring that all materials are prepared at the start of the course for 
students to prepare themselves and move at the pace that they find 
necessary to self-regulate throughout the course. 

Although there was little direct correlation between student feedback 
and high performance on the project parts, findings did show that 
students who did not seek a lot of feedback from the instructor early 
on had a difficult time grasping the concepts necessary to perform well 
on projects. This suggests that student self-regulation is an essential 
component to success in PBL (Toip & Sage, 2002; Barrows, 1996). 
Students need to actively seek out the instructor’s feedback and 
support when they perceive they are having difficulties. They can do 
so by asking questions, submitting project parts for instructor review, 
and asking for clarification of concepts or criteria. In this study, 
Blackboard personal or team areas, Blackboard discussion forums, 
email, whole class discussions, and team meetings (either virtual or 
face to face) were all utilized to communicate with the instructor for 
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these purposes. Some teams, as indicated in the previous section, 
accessed these options more often than others. Some teams received 
a higher frequency of feedback and performed well on tasks, while 
some sought out very little feedback and also performed well. The 
latter teams perhaps did not need as much support and were able 
to self-regulate through in-class discussions, the feed-up strategies 
given, and prior knowledge brought to the course. 

The social aspects of learning provided interactions among peers 
and the instructor that were critical to the development and the 
understanding of course content (Richardson & Swan, 2003). The 
social setting of this course varied from student to student. Some 
students were working from home and attending class virtually 
through the WebEx teleconferencing system while others were 
attending class face to face. Some teams were a mixture of both 
environments with one team member on campus and the other 
attending virtually. During class discussions, students from both 
learning environments contributed to the conversations and shared 
about their projects and learning. It was clear that students felt 
comfortable sharing ideas and providing input. These authentic 
evaluative experiences in which the instructor and peers can provide 
feedback and input made the social interaction meaningful (Ertmer 
et al., 2007) and improved overall satisfaction (Richardson & Swain, 
2003). One student at a distance pointed out that being in class could 
have been beneficial to the overall course experience, suggesting 
that a face-to-face environment is preferred by some students. This 
did not, however, interfere with the student’s overall performance 
on the course projects. Students also agreed that team meetings 
and class discussions, both social forms of instruction, contributed 
effectively towards overall success in this course. 

Future Research in Problem-based Learning and Formative 
Feedback 

This study brings to light the importance of formative feedback 
in problem or project-based learning. It is necessary that all forms 
including feed-up, feedback, and feed-forward be present and 
continue to support student success throughout the entire course. 
Characteristics of feedback should include frequency and timeliness 
which support individual or team needs, motivating feedback, 
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informative or specific feedback, and metacognitive questioning. 
These characteristics should also be delivered in various modes such 
as written, verbal, and dialogic; and through various methods such 
as one-on-one, peer or small-group, and self-assessment. 

Further research is necessary to address the areas of students’ 
individual differences, specifically in how they react to and utilize 
the feedback model. Future research should also explore how to 
prepare students for the disposition necessary to receive feedback. 
Another area for future research is the importance of determining 
student interests and abilities through pre-assessment to adjust 
feedback strategies such as frequency, timeliness, and specificity 
based on individual differences. Other areas of research that can 
help develop the idea of feedback in PBL include the student self¬ 
regulation necessary in this approach. This study concluded that a 
fair amount of self-regulation is necessary in order to be successful 
in PBL, especially knowing how to use and when to ask for 
feedback. Further studies could also focus on why some students 
choose to seek more feedback, how to support students who seek 
more feedback, and how to teach students to understand when they 
should request for help or clarification. Another important factor to 
consider is the length of time required by instructors to deliver this 
multifaceted feedback. Dialogic feedback, informative and specific 
written feedback, and one-on-one team meetings appear to be the 
some of the most desirable characteristics implemented. 
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Appendix A 

Synthesis of the literature on forms, functions and features of 
feedback 


Feedback Description Definition 

Forms of Written Written feedback is a genre in which the desire 

Feedback is to give clearly communicated messages to the 

learner in a written form. When using this mode of 
feedback, it is necessary to consider the tone and 
word choice as other social cues are not present 
to convey meaning. It is suggested that this type 
of feedback tends to work better in summative or 
graded assignments as students often look straight 
to the grade and may not pay close attention to the 
specific feedback (Brookhart, 2008). 

Oral Oral feedback is perhaps one of the most 

important forms of feedback when considering 
the relationship between the instructor and the 
student. It often occurs informally during real¬ 
time observation of a student’s work or formally 
during one-on-one conferencing (Brookhart, 
2008). Oral feedback can be given during class 
discussions to correct whole group misconceptions 
or can be used to immediately praise or correct an 
individual student’s performance. This real-time 
feedback allows for a quick and timely response 
that is often necessary during the acquisition of 
difficult learning tasks (Corbett & Anderson, 
2001 ). 

Dialogic The limitations of simply providing one way 
written or oral feedback leave the learner with room 
for incorrect interpretation, ignored suggestions, 
and no way to ask questions or respond. Distance 
and mass education settings also pose a problem 
for the relationship between student, instructor, 
and peers. Dialogic feedback provides a format 
for interactive exchange between the learner 
and the instructor/tutor, or the learner and peers. 
This approach involves a two-way conversation 
which can include how well a student is meeting 
quality standards specific to assignment criteria, 
what good performance will look like, and how 
the learner can proceed from this point forward 
(Careless, Salter, Yang, & Lam, 2011). 


(continued) 
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Feedback Description 

Definition 

Frequent 

■ 

5 

) 

■ 

Early research in this area suggests that when 
a learner has a degree of awareness of their 
performance, a tasks output increases (Hundal 
1969). Thus, frequent feedback during early 
acquisition of learning could motivate and guide 
the learner to improve performance (Salmoni, 
Schmidt, & Walter, 1984). Anderson, Kulhavy, 
& Andre (1971) argued that those learners who 
receive more frequent feedback perform better 
than those who receive none. However, later 
research examining the effect of frequent feedback 
(Salmoni, Schmidt, & Walter, 1984; Lam, 
DeRue, Karam, & Hollenbeck, 2011; Chhokar 
& Wallin, 1984) concluded that feedback given 
too frequently is not beneficial and can degrade 
learning causing the learner to divert more 
cognitive resources toward self-regulation rather 
than on-task activities such as learning (Lam, 
DeRue, Karam, & Hollenbeck, 2011). 

■ Timely 

■ 

■ 

L 

The timing or efficiency of feedback is another 
characteristic that has conflicting viewpoints 
throughout the literature. The research argues 
the benefits and consequences of both delayed 
and immediate feedback and points out factors 
that may impact these outcomes (Kulhavy & 
Anderson, 1972; Phye & Andre, 1989; Mathan 
& Koedinger, 2003; Clariana, 1999; Schmidt, 
Young, Swinnen & Shapiro, 1989; Corbett & 
Anderson, 2001). Immediate feedback describes 
any feedback that is given right after the student 
has responded to the task at hand. This method of 
delivering feedback can be extremely effective in 
the initial stages of learning (Schmidt et al., 1989), 
especially in areas which require complex problem 
solving skills (Corbett & Anderson, 2001). 

Functions of Directive 

Feedback 

Directive feedback tells the student what needs 

to be fixed or revised. Such feedback tends to be 
more specific (Black & William, 1998). 

Facilitative 

Facilitative feedback provides comments and 
suggestions to help guide students in their own 
revision and conceptualization (Black & William, 
1998). 


(continued) 
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Feedback Description Definition 

Specific Feedback specificity is referred to as, “the level 
of information presented in feedback messages” 
(Goodman, Wood, & Hendrickx, 2004, p. 
248). Feedback can be considered general and 
facilitative in nature, or specific and have more 
elaborative and direct qualities. Much like 
frequency, the specificity with which feedback 
is given has many variables which can influence 
its effects. For example, a high level of specific 
feedback in initial learning is beneficial for 
improved performance because students are given 
direct information with which they can use to 
shape a foundation of understanding (Goodman, 
Wood, & Hendrickx, 2004). 


Sources of 
Feedback 


Instructor’s Feedback from the instructor and expert of course 
one-on-one content is often preferred by learners, especially 
feedback those that view high achievement as valuable 
(Vancouver & Morrison, 1995). When seeking 
feedback, those that are motivated to learn want 
to be sure they are obtaining reliable and accurate 
information and feel this best comes from source 
experts (Vancouver & Morrison, 1995). For this 
feedback to support the learner it must be clearly 
related to the goals or criteria, help the student 
take action to reach those goals, and must use an 
instrument that delivers the feedback in a way 
which positively supports the learner (Nicol & 
Macfarlane-Dick, 2006). Furthermore, building a 
positive and supportive relationship between the 
instructor and student is essential to the one-on- 
one learning process (Cramp, 2011). 


Self- Self-assessment can be defined as an application 

assessment of many distinct processes including the selection 
of appropriate external data and standards, an 
awareness of one’s own current state, and the 
critical reflection of one’s own performance 
(Sargeant et ah, 2010). Epstein, Siegel, and 
Silberman (2008) suggest that self-assessment 
lies in two major domains which include 1) “the 
integration of high quality external and internal 
data to assess current performance” and 2) “the 
capacity for ongoing self-monitoring during 
everyday practice” (p. 11). When students are 
provided with an authentic evaluative experience 
in learning, the evaluative decisions begin to shift 
in responsibility from teacher to learner (Sadler, 
1989). 


(continued) 






http://mjli.uum.edu.my 


Malaysian Journal of Learning and Instruction: Vol. 13 No. 2 (2016): 1-41 


Feedback Description 

Definition 

Peer and 

Students begin to learn and/or apply a variety of 
metacognitive and self-evaluative skills which 
will be critical in future professional and non¬ 
professional roles (Boud, Lawson, Thompson, 
2013). Essential are self-monitoring skills such 
as noticing, self-questioning, and suspending 
personal judgments while assessing one’s own 
work (Epstein, Siegel, & Silberman, 2008). 

“In the social learning system, new patterns 


Instructor’s of behavior can be acquired through direct 
Small Group experience or by observing the behavior of others” 


■ Feedback 

(Bandura, 1977, p. 3). 

■ 

5 

) 

■ 

Peer and small group feedback are derived from 
this social learning theory developed by Albert 
Bandura (1977). Instructional design, especially 
that of online environments, should incorporate 
social aspects of learning (Richardson & Swan, 
2003). Social learning strategies allow the 
instructor to share some of the responsibility of 
the learning with the learners (Ertmer, Richardson, 
Belland, & Camin, 2007) and provide an 
opportunity for learners to construct knowledge in 
collaboration with peers. 
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Appendix B 

Examples of the instructor’s verbal/dialogic feedback 


Characteristics of Verbal/Dialogic Feedback 

Feedback 

Motivating • “You brought up a very good point.” 

• “Right, so you want your learner to demonstrate . . .” 

• “Excellent, so what you described is . . . and that is how 
you created your goal.” 

• “I think your model is very appropriate because . ..” 

• “At this point you may be struggling but that is ok.” 

Feed-Forward Goals • “Your next step would be to start calculating how much 
it is going to cost you.” 

• “You need to now do the plan based on what you estab¬ 
lished your goals were.” 

• “Keep thinking about your goal, and make sure you are 
assessing it properly.” 

• “You have done the needs analysis, now start expanding 
on that plan for the needs and goals you identified.” 

• “In addition to human performance, you may want to 
look at the procedure.” 

Informative/Specific • "Wait on creating phases until your goals are formu¬ 
lated.” 

• “Be careful not to exclude parts that do not apply to you 
when learning.” 

• “You need to make sure you are putting emphasis on 
the outcome.” 

• “Use terminology that is a bit more appealing or attrac¬ 
tive to the reader.” 

• “If you take a look at my feedback, you can see how 
to differentiate between benchmarks and instruments.” 

Metacognitive • “Is there a process that should take place?” 

• “[Y]ou basically have two target groups of people that 
would have to master skills and knowledge, so what are 
your 2 goals?” 

• "What should you expect after one year?” 

• “Do you train the trainer at the beginning or do you fol¬ 
low up?” 

• “Did anything puzzle you during the process?” 
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Examples of the instructor’s written feedback 


Characteristics of Written Feedback 
Feedback 

Motivating • “Good, so the change is for the whole school?” 

• “Good job of explaining the forces for the change.” 

• “Very good! The purpose of the proposed change is very 
clear.” 

• “Good, you have some estimation of the resources you 
need and where the funding is coming from.” 

• “Very good!! Excellent decision!” 

Feed-Forward Goals • “The means of implementing the change will have to be 
defined later, and you have to explain what the stages of 
the whole school adoption are.” 

• “Do you want to include an evaluator who could track 
the progress you are making?” 

• Make the scope of the change clear. How this phased 
approach expands over time and what is the final scope?” 

• “... indicate if are enough resources internally to provide 
needed training for staff. What will be developed and 
how will be an implementation/ planning issue that will 
be decided on later.” 

• “You may want to include the purposes or goals targeted 
for this phase.” 

• “Consider using the model to explain how it will inform 
your planning and implementation of the change.” 

Informative/Specific • “Problem-based is not the same as project-based 
learning. If project-based learning is your focus use 
project-based, not problem-based.” 

• “Always start with specific goal and objectives.” 

• “I would have liked to see a project summary page that 
highlighted the project core ideas and plans. . . .” 

• “You need to make the impact of this change on the 
organization outcome (goals) clearer so that the value 
and impact of the change are measurable.” 

• "While your argument for a good match between the 
organization conditions and the model is established, it 
is hard to know how you would apply the change model 
to plan the change.” 

Metacognitive • “How about curriculum and instructional materials? Do 

they have to change? How about scheduling and time 
for classes?” 

• “Ok, then how will it expand?” 

• “Should this be changed to benchmark?” 

• “How would you use the model to plan and implement 
the change?” 

• “How many new positions and experienced individuals?” 






